Repository navigation
feat(topology): build the service map from whole-window ClickHouse trace aggregates - #250
Conversation
…ace aggregates The ClickHouse trace source joined the newest 1,000 spans of a ~160k-span hour, so most parent/child pairs fell outside the sample: low-volume calls and databases dropped out and most spans showed as orphans. - ClickHouseOtelAggregateSource reads the catalog's exporter-schema table with three bounded read-only queries: service/kind/status attribute groups with span counts, parent->child service pairs and span-link pairs (1,000-row limit each; a saturated limit is cap_reached, any failed query fails the read). Generated/custom templates and Tempo keep span sampling. - buildTraceGraph weights aggregate rows and draws source-counted pairs as span evidence on the same service nodes. - Service-graph metric rows keep connection_type: database/messaging servers become db/queue nodes only without span evidence, and virtual_node rows draw the uninstrumented side as an external node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🤖 AI Code Review (two independent reviewers)Reviewer responses: codex/ALL claude/ALL Status: PASSED — No blocking issues found All verification is done. Both panel claims are now reconciled against BASE + patch; I can write the chair synthesis. Chair Synthesis — PR #250: whole-window ClickHouse trace aggregates1. SummaryThe PR replaces the newest-1,000-span sample with three bounded read-only ClickHouse aggregate queries (node groups, parent→child pairs, span links) for catalog exporter-schema sources, adds weighted span rows and source-counted call edges to the graph builder, rewrites node-cap selection to keep edge endpoints together, and filters Tempo service-graph rows by Chair verification performed: 2. Issues per lensL2 — Code correctnessNo CRITICAL/MAJOR adopted.
L3 — Security / AWS mutation safetyNo findings — both models agree, and the chair independently confirmed. Every new statement is a SELECT; no IAM, Terraform resource, or mutation surface is touched ( L4 — Observability / data-integration correctnessNo CRITICAL/MAJOR adopted. One model (claude) flagged five MAJORs here; codex found nothing. The chair verified each claim's facts but downgrades all five on severity:
L5 — Docs / ADR consistencyNo CRITICAL/MAJOR. No ADR bodies exist in this tree to diff (only ADR-023); the mutation-freeze invariant is untouched. CHANGELOG follows the repo rule: one feature bullet per section, no PR/round numbers, EN/KO 1:1. New runbook/api-reference content is English-only as required. Verified-accurate: caps (200/500/1,000), query counts, reason taxonomy, exclusion wording, and the runbook's test command and cross-link anchor.
3. Suggestions
4. VerdictCodex's clean report and claude's findings converge on the only gate-relevant questions — read-only safety and the absence of missing-definition/activation defects — which the chair independently confirmed. Claude's six MAJORs are each factually grounded but, on chair verification against BASE and the resulting code, all land as disclosed design tradeoffs, internal-ranking effects, or missing-disclosure plumbing rather than incorrect published data, security exposure, or policy violations. No CRITICAL or MAJOR finding survives verification; the image manifest lists no images. IMAGE_COVERAGE: NOT_REQUIRED Triggered by commit |
- Bump graph CATALOG_VERSION to v4 so indexed Prometheus/Mimir instances rebuild and pick up the connection_type PromQL. - Defer a metric database/queue row only when trace spans show the same client service using a database/queue, instead of whenever any span exists; count deferrals as deferredMetricCalls. - Accept the aggregate path only for exactly the bare exporter columns, so generated or aliased otel_v1 templates keep the span reader. - Push the sorted identity map into both join sides (bounded hash build) and mapSort grouped maps so equal attribute sets group together. - Keep ARN service.instance.id claims so claimedInstance can still veto them. - Require empty proof from every aggregate; guard link-array lengths before ARRAY JOIN; sum durations in milliseconds within safe integers. - Metric-derived db nodes carry no account/region attribution; metric volume ranks nodes under the node cap with external peers ranked like services. - Document the two-plus-optional-links queries, caps and boundary pairs, and the runbook aggregate contract; add an external node color. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…coped - Reduce host.id/service.instance.id to EC2 IDs/ARNs and any other cloud.resource_id to one 'unbindable' marker, so host UUIDs stop splitting groups while claimedInstance keeps its cloud.resource_id veto. - Treat a metric database/queue row as a duplicate only when a span from a compatibly scoped client of the same name names that dependency; other dependencies and scopes keep their metric rows. The internal counter is no longer documented as published metadata. - Rank label-only metric db/queue nodes by volume with services so a high-cardinality metric cannot evict service nodes. - Clamp aggregate rows to the limit, disclose out-of-window rows, cap merged workload pods and bound metric node names. - Docs: distinguish failed envelopes from skipped rows, qualify counts by cap_reached, and move the mocked suites out of the PostgreSQL block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…des by edge, fall back to sampling - Service-graph metrics contribute service->service calls only, as graph-store's metrics-source contract already states. Rows Tempo marks database, messaging_system or virtual_node are not drawn (they are not services, and their label-only nodes would carry no provenance through the agent SQL reader view); databases and queues come from trace spans with host-qualified identity. This removes the metric db/queue/external node kinds and the span-based deferral. - Trace node-cap selection takes the heaviest edges' endpoints first, so a kept db/queue/workload always keeps its caller and many low-volume dependencies cannot evict their services. - A failed aggregate query falls back to the bounded newest-span reader for that rebuild (partial, incomplete_collection) instead of retaining a frozen graph; a malformed envelope still fails the read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
The service map's ClickHouse source read the newest 1,000 spans (
ORDER BY Timestamp DESC LIMIT 1000). The dev datasource has about 164k spans per hour, so that sample covers roughly the last 20 seconds. Acallsedge needs both the parent and the child span inside the sample, so most pairs were lost: low-volume calls and databases dropped out of the map and most spans showed as orphans.The Prometheus service-graph metrics (
traces_service_graph_request_total) were grouped only byclient/server. That turned database and queue servers (for exampleadmin,mall,redis) and Tempo'suservirtual node into fake service nodes.What changes
ClickHouseOtelAggregateSource(web/lib/trace-source.ts)otel_v1template, or from the unregistered default.cap_reached. Any failed query fails the whole read, so nodes never publish without their edges.buildTraceGraph(web/lib/trace-graph.ts)queries/runs_on/messaging edges, and the instance↔DB observation count and first-seen time.evidence: 'trace') as span evidence on the same service nodes, with no self-loops.connection_typeis kept in the catalog PromQL (scripts/v2/workers/graph_catalog.py) and parsed in the web code.databaseandmessaging_systemrows becomedb/queuenodes only when no span evidence exists. Spans carry the host-qualified identity and would otherwise duplicate the same database under another name.virtual_noderows draw the uninstrumented side as anexternalnode.graph-store.tspasses the aggregated edges through.graph-sources.tspicks the aggregate source for catalog templates and for the default fallback.docs/api-reference.md§Trace graph sources) and a bilingual CHANGELOG entry.Verification
npx tsc --noEmit -p .is clean. Full webnpm test: 372 files, 5,718 tests passed.pytest scripts/v2/workers/test_graph_catalog.py: 24 passed.New tests cover:
; DROPsuffix and non-catalog SQLcap_reached, malformed rows and empty-proofconnection_typenode kindsLive check against the dev ClickHouse (datasource 3,
otel.otel_traces, read-only, throughawsops-dev-agent-clickhouse-mcp). I ran the exact SQL the code generates:Fed through the new mapper and builder, those rows produce: status
ok, 0 orphan spans, 0 invalid spans, 71 nodes (31 service / 28 workload / 12 db) and 144 edges (62 calls / 54 queries / 28 runs_on). An example edge iswishlist → product-catalogwith 2,204 calls.Rollout notes
datasource_diagnosis_enabled=false), the default fallback is the aggregate source. It relies on the datasource's configured database, which isotelon dev.graph_catalog.pychange ships in the worker Lambda zip, so it needs the usual Terraform plan/apply. Catalog rows pick upconnection_typeon the nextdatasource_indexrun. Until then the old PromQL simply lacks the label and behaves as before.🤖 Generated with Claude Code